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In a real sense, PISA 2003 has touched the mathematics education community by 
stealth rather than by storm. Although PISA brings “baggage ” commonly associated 
with international assessments, it takes some refreshing perspectives especially in the 
way that it envisions and assesses mathematical literacy. In this panel diseussion we 
focus on some of the issues associated with PISA: scrutiny of student performance, 
construct and consequential validity, what makes items difficult for students and the 
potential impact of PISA on mathematics education research. In selecting these 
issues we merely begin the debate and open the way for your participation. 

WHAT IS PISA? 

The Programme for International Student Assessment ([PISA], OECD, 2005) is an 
international standardized assessment in reading literaey, mathematieal literaey, 
problem-solving literaey and scientifie literaey. It started in 1997 when OECD 
countries began to collaborate in monitoring the outcomes of education and, in 
particular, assessed the performance of 15-year-old school students according to an 
agreed framework. Tests have typically been administered to 4,500-10,000 students 
in each country. The first assessment in 2000 which focused mainly on reading 
literacy surveyed students in 43 countries while the second assessment in 2003 
involved 41 countries and focused mainly on mathematics and problem solving. The 
third assessment in 2006 will largely emphasize scientific literacy and is expected to 
include participants from 58 countries. In this panel discussion we will concentrate 
on PISA 2003 and those aspects of it that deal with mathematical literacy. 

THE PISA MATHEMATICAL LITERACY ASSESSMENT 

In describing their approach to assessing mathematical performance, PISA 
documents (e.g., OECD, 2004a) highlight the need for citizens to enjoy personal 
fulfilment, employment, and full participation in society. Consequently they require 
that “all adults-not just those aspiring to a scientific career-be mathematically, 
scientifically, and technologically literate” (p. 37). This key emphasis is manifest in 
the PISA definition of mathematical literacy: “ ...an individual’s capacity to identify 
and understand the role that mathematics plays in the world, to make well-founded 
judgments and to use and engage with mathematics in ways that meet the needs of 
that individual’s life as a constructive, concerned, and reflective citizen” (OECD, p. 
37; see also Kieran, plenary panel papers). 

Reflecting this view of mathematical literacy, PISA documents (e.g., OECD, 2004a) 
note that real-life problems, for which mathematical knowledge may be useful, 
seldom appear in the familiar forms characteristic of “school mathematics.” The 
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PISA position in assessing mathematies was therefore designed “to eneourage an 
approaeh to teaehing and learning mathematies that gives strong emphasis to the 
proeesses assoeiated with eonfronting problems in real-world eontexts, making these 
problems amenable to mathematieal treatment, using the relevant mathematieal 
knowledge to solve problems, and evaluating the solution in the original problem 
eontext” (OECD, 2004a, 38). In essenee, mathematical literacy in the PISA sense 
places a high priority on mathematical problem-solving and even more sharply on 
mathematical modelling. 

Although PISA’s devotion to mathematical modelling has my unequivocal support, 
my experience tells me that it is not easy to incorporate effective mathematical 
modelling problems in a test that has fairly rigid time constraints. In addition, 
although the term mathematical modelling is relatively new in school mathematics 
(Swetz & Hartzler, 1991), there are instances of mathematical modelling even in the 
notorious public examinations of more than 50 years ago. I well remember the 
following problem in an examination that I took in 1953. It seems to me that it is a 
genuine modelling problem and it was certainly not a text book problem or a problem 
that anyone of that era had practised. Moreover, the fact that less than 10% percent of 
the 15 to 16-year-old students taking the examination solved the problem is both deja 
vu and prophetic for those setting the directions for the PISA enterprise. 

In a hemispherical bowl of radius 8 inches with its plane section horizontal stands water 
to a depth of 3 inches. Through what maximum angle can the bowl be tilted without 
spilling the water? Give your answer to the nearest degree (University of Queensland, 
1953) 

Accordingly, even though members of our panel valued the PISA emphasis on real- 
world problems and mathematical modelling, there was no shortage of issues to 
debate. In particular, there were issues about the framework, the validity of the 
assessment, the construction of items, the measurement processes, the conclusions 
and the interpretations especially interpretations that cast the findings into the realm 
of an international “league table”. Consequently, we faced a problem in selecting 
which issues to examine. Let me presage the papers of the other panellists by 
providing an entree of the issues that reverberated over our internet highways. 

WHAT ISSUES DOES PISA RAISE FOR MATHEMATICS EUDCATION? 

As the conference theme was learners and learning we questioned whether PISA 
assessment really was designed to support a real-world approach to mathematics 
teaching and learning. We also raised questions about whether student performance 
in the PISA assessments mirrored student performance in other mathematics 
education research on learning and teaching. Although appropriate data was not 
easily accessible, we wondered what the PISA study told us about patterns of 
classroom activity in different cultures. Yoshinori Shimizu (plenary panel papers) did 
examine this from a cultural perspective by scrutinizing Japanese students’ responses 
to some PISA items. 



PME29 — 2005 



1-72 





Jones 



Issues assoeiated with item validity, item authentieity, and item diffieulty were 
consistently part of our discussions. The “triangular park problem” (see Williams, 
plenary panel papers) was hotly debated and members of the team even spent 
considerable time looking for triangular parks or car parks. This was part of our 
conversation on real world or authentic assessment and this issue is taken up further 
by Julian Williams under the broader topic of construct validity. Carolyn Kieran (see 
plenary panel papers) takes up the issue of “what makes items difficult for students?” 
She observes that the dijficulty levels of some PISA items are problematic and raises 
doubts about how much we know about what students find difficult in certain 
mathematical tasks. 

The politics of international assessment studies like PISA (OECD, 2004a) and Trends 
in International Mathematics and Science Study ([TIMSS], Mullis et al., 2004) were 
high on our debate list. Not only do these debates raise highly volatile issues and 
national recriminations, they also generate profound questions for those countries that 
are doing well and for those who are not. In addition to issues that focus specifically 
on the international league, assessment studies like PISA produce a range of related 
debates about factors such as gender, ethnicity, socio-economic status, systemic 
characteristics, approaches to learning, student characteristics and attitudes, and of 
course fiscal support (OECD, 2004b). Julian Williams (see plenary panel papers) 
tackles a number of these political issues especially those related to accountability: 
managing targets, dealing with league tables, and performance-related reviews. 

There was considerable interest in discussing the impact of international assessment 
studies on mathematics education research. At the forefront of such issues is the 
question: What does PISA say to researchers interested in assessment research? 
Yoshinori Shimizu (see plenary panel papers) will talk about this more specifically as 
he refers to the benefits that can be gleaned by researchers through an examination of 
PISA’s and TIMSS’s theoretical frameworks, methodologies, and findings. Eor 
example, he notes that the detailed item scales and maps in PISA will enable 
researchers to perform a secondary analysis of students’ thinking and accordingly 
gain a deeper understanding of learners and learning. Michael Neubrand (see plenary 
panel papers) also looks at the potential of PISA to stimulate research in mathematics 
education. He focuses on the structure of mathematical achievement especially in the 
way that PISA conceptualizes achievement through the aegis of a mathematical 
literacy framework. This gives rise to an interesting dialogue with respect to both 
individual and systemic (collective) competencies in mathematics and how they can 
be measured. There are of course other important questions such as “What do studies 
like PISA say to mathematics education researchers about methodological issues such 
as qualitative versus quantitative research?” Although this particular question is not 
directly addressed, the panel refers frequently to methodological issues and as such 
issues a challenge to the participants for further engagement and debate. 
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CONCLUDING COMMENTS 

I believe that this panel diseussion is most timely as I am not eonvineed that 
mathematies edueators are as eognizant as they might be about the impaet of the 
burgeoning industry that eneompasses international studies like PISA (OECD, 2003) 
and TIMSS (Mullis et ah, 2004). Although the build up and dissemination of PISA 
has been slow to take root in the mathematies education research community, the 
findings have certainly not gone unnoticed by national and state governments, 
educational systems, business leaders and parent groups. They know where their 
nation or their state came in the “league stakes” but they have little understanding of 
the intent and limitations of such studies. Accordingly, an important aim of this panel 
is to encourage mathematics education researchers to be more proactive not only in 
publicly illuminating and auditing research like PISA but also in identifying ways in 
which PISA can connect with and stimulate their own research. In the words of Sfard 
(2004, p. 6) we should exploit these special times in mathematics education: 

Confronting the broadly publicized, often disappointing, results of the international 
measurements of students’ achievements, people from different countries started 
wondering about the possibility of systematic, research-based improvements in 
mathematics education 
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